Theoretical Quiz — Module 2
Q1. What are the five main components of Airflow's architecture? Describe the role of each.
Show Answer
- Scheduler — Parses DAGs, creates DAG Runs, and schedules tasks for execution
- Webserver — Serves the UI for monitoring and managing workflows
- Executor — Determines how tasks are distributed (locally, via Celery, or K8s)
- Metadata Database — Stores all state, configuration, connections, and XCom data
- Triggerer — Handles deferrable operators asynchronously
Q2. Why should you never use SQLite in production?
Show Answer
SQLite doesn't support concurrent writes. In production, the Scheduler, Webserver, and Workers all write to the database simultaneously. SQLite would cause lock contention, data corruption, and unreliable behavior. Use PostgreSQL or MySQL instead.
Q3. Explain the difference between CeleryExecutor and KubernetesExecutor. When would you choose each?
Show Answer
- CeleryExecutor: Tasks are distributed to pre-provisioned workers via a message broker (Redis/RabbitMQ). Workers are always running. Best for stable workloads with predictable resource needs.
- KubernetesExecutor: Each task spawns a new Kubernetes pod with custom resources. Pods are destroyed after completion. Best for variable workloads, strict task isolation, and cost optimization.
Choose CeleryExecutor for consistent, high-throughput pipelines. Choose KubernetesExecutor for elastic, cloud-native deployments with varying resource requirements.
Q4. What is a deferrable operator and why was the Triggerer component introduced?
Show Answer
A deferrable operator can "pause" its execution and hand off monitoring to the Triggerer. The Triggerer uses Python's asyncio to efficiently monitor hundreds of waiting conditions without consuming worker slots.
Without deferrable operators, a sensor waiting for a file on S3 would occupy a worker slot for the entire wait time (potentially hours). With deferrable operators, the worker is freed immediately, and the Triggerer handles the waiting asynchronously.